Non-Deterministic Parallelism Considered Useful

The development of distributed execution engines has greatly simplified parallel programming, by shielding developers from the gory details of programming in a distributed system, and allowing them to focus on writing sequential code [8, 11, 18]. The “sacred cow” in these systems is transparent fault tolerance, which is achieved by dividing the computation into atomic tasks that execute deterministically, and hence may be re-executed if a participant fails or some intermediate data are lost. In this paper, we explore the possibility of relaxing this requirement, on the premise that non-determinism is useful and sometimes essential to support many programs. In recent years, the class of problems that distributed execution engines can solve has grown. The original MapReduce system addresses problems that can be decomposed into embarrassingly-parallel map and reduce phases [8]. Dryad extends MapReduce by supporting arbitrary directed acyclic graphs (DAGs) of tasks [11]. We recently developed a system called CIEL, which supports arbitrary DAGs in which a task may spawn a subgraph of tasks, hence enabling unbounded iteration [18]. Therefore, the set of all CIEL jobs is a strict superset of all Dryad jobs, which in turn is a strict superset of all MapReduce jobs. We have previously argued that CIEL is universal, since it can execute jobs that are specified in a Turing-complete language [17]. However, this universality merely implies that any algorithm can be implemented on CIEL; it does not necessarily follow that the implementation will be efficient. Deterministic execution is a conservative requirement, rooted in the assumption that the participants in a distributed execution engine fail frequently. We briefly discuss the fault tolerance benefits of determinism in Section 2. While this assumption may be true for the largescale clusters on which MapReduce and Dryad were originally developed, it is less obvious for the smaller clusters on which they are often deployed [1]. Our high-level argument is that deterministic execution should not be a requirement at the system level. Deterministic parallelism can be implemented on a system that allows non-determinism [4]; non-deterministic parallelism cannot be implemented on a system that provides only deterministic abstractions [3]. As we show in Section 3, there are several applications that benefit from non-determinism, and we believe that it would be useful to extend the benefits of distributed execution engines to these applications. Furthermore, in our proposed approach (described in Section 4) programs would be “deterministic by default” [5]; we simply extend the set of abstractions in CIEL to allow explicit non-determinism where desirable. If one must limit an individual job to deterministic execution, this can be enforced at the language level, or with a policy in the underlying engine. In this position paper, we make three contributions: